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DETAILED ACTION 

1 . This communication is responsive to Amendment A, filed 04/20/04. 

2. Claims 1-13, 15-28, and 30-35 are pending in this application. In Amendment A, claims 
1, 6, 7, 9, 10, 12, 17, 22, 26, 27, 28, 30, and 31 are independent claims, and claims 14 and 29 are 
cancelled. This action is made non-final. 

Claim Rejections - 35 USC §102 

3. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the 
basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(e) the invention was described in (1) an application for patent, published under section 122(b), by another filed 
in the United States before the invention by the applicant for patent or (2) a patent granted on an application for 
patent by another filed in the United States before the invention by the applicant for patent, except that an 
international application filed under the treaty defined in section 351(a) shall have the effects for purposes of this 
subsection of an application filed in the United States only if the international application designated the United 
States and was published under Article 21(2) of such treaty in the English language. 

4. Claims 1, 4, 5-7, 9, 10, 12, 15, 16, 22, 24, 25, 26-28, and 30 are rejected under 35 
U.S.G. 102(e) as being anticipate by Kuzunuki et al. (U.S. Patent No. 5,917,490). 

As to claim 1, Kuzunuki teaches an information input/output system to be used for user 
operations relating to an object in an information space realized by expanding a digital space of a 
computer into the real world, said system comprising: 

one or more than one operation surfaces arranged in said information space (display 101, 
col. 6 lines 14-32, block 400, col. 6 lines 44-58, and fig. 1); 

a display means for displaying an image on said operation surfaces (e.g., col. 6 lines 14- 
32, projectors 105 and 107, col. 7 lines 11-29, and fig. 2); 
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an imaging means for picking up an image of said operation surfaces (e.g., col. 2 lines 
41-51, overhead camera 300 to pick up actual images, col 6 lines 1-12, and figs. 1-2); 

a stationary environment type computer arranged in said information space (e.g., col. 6 
lines 1-32, lines 44-58, and figs. 1-2); and 

one or more than one physical objects mounted on said operation surfaces and containing 
a visually identifiable visual marker on the surface thereof (a barcode is assigned to an objects, 
col. 21 lines 20-26, lines 42-58); 

said environmental type computer being adapted to execute (a program which is executed 
by the information processing unit, col. 19 lines 41-65); 

a processing operation of recognizing the identification information and the position 
information of each of said physical objects in said visual marker (e.g., col. 21 lines 20-58); 

a processing operation of recognizing the digital object dropped to a site on the surface of 
each of said physical objects (e.g., col. 7 lines 46-65 and col. 14 lines 6-9); and 

a processing operation of forming link information for linking the digital object to the 
dropped site on the surface for each of said physical objects (e.g., col. 3 lines 44-52, link between 
physical objects and electronically stored information, col. 19 lines 36-54). 

As to claim 4, Kuzunuki teaches the information input/output system according to claim 
1, wherein 

said imaging means can identify the position indicated by an optical pointer for indicating 
a specific position by irradiating a beam of light with a predetermined wavelength (using infrared 
light, col. 21 lines 20-26); and 
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the user is allowed to indicate a position in said information space by means of 
coordinates and the optical pointer (a barcode reader, col. 21 lines 20-26). 

As to claim 5, Kuzunuki teaches the information input/output system according to claim 
1, wherein said environment type computer executes a processing operation of calling the linked 
digital object and/or displaying the digital object to the user in response to a user operation 
applied to the site of forming the link information on the surface of each of the physical objects 
on the basis of the image picked up by said imaging means (e.g., col. 3 lines 44-52 and col. 19 
lines 41-65). 

As to claims 6-7, they are similar in scope to claim 1 above; therefore, rejected under 
similar rationale. 

As to claim 9, Kuzunuki teaches an information input/output system to be used for user 
operations relating to an object in an information space realized by expanding a digital space of a 
computer into the real world, said system comprising: 

one or more than one operation surfaces arranged in said information space; a display 
means for displaying an image on said operation surfaces; an imaging means for picking up an 
image of said operation surfaces; a stationary environment type computer arranged in said 
information space; and one or more than one physical objects mounted on said operation surfaces 
and containing a visually identifiable visual marker on the surface thereof (note the rejection of 
claim 1 above); 

at least one of said physical objects being a virtual camera having a virtual imaging 
direction as specified by the position information contained in the visual marker (e.g., the 
facsimile direction to be placed within a given distance, col. 2 line 66-col. 3 line 11). 
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As to claim 10, Kuzunuki teaches the information input/output system according to claim 
9, wherein 

said environmental type computer is adapted to execute (e.g., col. 19 lines 41-65); 

a processing operation of recognizing/identifying said virtual camera and identifying the 
position information and the virtual imaging direction (a video camera 6 mounted above a desk 
surface 2 for picking up physical documents, col. 6 lines 1-7, and fig. 1); 

a processing operation of generating a virtual picked up image according to the position 
information and the virtual imaging direction (e.g., col. 2 line 66-col. 3 line 1 1, and col. 10 line 
56-col. 1 1 Hne 27); and 

a processing operation of displaying the virtual picked up image to the user (figs. 6A-8B). 

As to claim 12, Kuzunuki teaches an information input/output system to be used for user 
operations relating to an object in an information space realized by expanding a digital space of a 
computer into the real world, said system comprising: 

one or more than one operation surfaces arranged in said information space; a display 
means for displaying an image on said operation surfaces; an imaging means for picking up an 
image of said operation surfaces; a stationary environment type computer arranged in said 
information space; one or more than one physical objects mounted on said operation surfaces and 
containing a visually identifiable visual marker on the surface thereof (note the rejection of claim 
1 above); and 

an optical pointer adapted to point a specific position by irradiating a beam of light with a 
predetermined wavelength (using infrared light, col. 21 lines 20-26); 
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said imaging means being capable of identifying the position pointed by said optical 
pointer (a barcode reader, col 21 lines 20-26). 

As to claim 13, Kuzunuki teaches the information input/output system according to claim 
12, wherein 

said environment image type computer executes on the basis of the picked up by said 
imaging means (e.g., col. 19 lines 41-65); 

a processing operation of recognizing the user operation on said operation surfaces, using 
said optical pointer (a barcode reader, col. 21 lines 20-26); and 

a processing operation of controlling the display of the digital object by said display 
means according to the result of the recognition (e.g., col. 21 lines 20-58). 

As to claim 15, Kuzunuki teaches an information input/output system to be used for user 
operations relating to an object in an information space realized by expanding a digital space of a 
computer into the real world, said system comprising: 

one or more than one operation surfaces arranged in said information pace; a display 
means for displaying an image on said operation surfaces; an imaging means for picking up an 
image of said operation surfaces; a stationary environment type computer arranged in said 
information space; one or more than one portable computers capable of exchanging digital 
objects with other computers; one or more than one physical objects mounted on said operation 
surfaces and containing a visually identifiable visual marker on the surface thereof (note the 
rejection of claim 12 above); and 



Application/Control Number: 09/739,952 Page 7 

Art Unit: 2179 

a portable ID recognition device adapted to recognize the visual marker on the surface of 
each of said physical objects and being capable of holding or releasing the recognized 
identification information (e.g., col. 21 lines 20-58). 

As to claim 16, Kuzunuki teaches the information input/output system according to claim 
15, wherein 

said environment type computer executes (e.g., col. 19 lines 41-65); 

a processing operation of obtaining a digital object corresponding to the held 
identification information in response to the holding operation of said ID recognition device 
(e.g., col 21 lines 20-58); and 

processing operation of transferring a digital object corresponding to the held 
identification information to a nearby physical object in response to the releasing operation of 
said ID recognition device (e.g., col. 21 lines 20-58). 

As to claim 22, it is individually similar in scope to claim 1 above; therefore, rejected 
under similar rationale. 

As to claim 24, it is individually similar in scope to claim 4 above; therefore, rejected 
under similar rationale. 

As to claim 25, Kuzunuki teaches the information input/output system according to claim 
22, flirther comprising: 

a step of calling the linked digital object and/or displaying the digital object to the user in 
response to a user operation applied to the site of forming the link information on the surface of 
each of the physical objects (e.g., col. 3 lines 44-52, Hnk between physical objects and 
electronically stored information, col 19 lines 36-54). 
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As to claim 26, it is individually similar in scope to claim 7 above; therefore, rejected 
under similar rationale. 

As to claim 27, it is individually similar in scope to claim 1 above; therefore, rejected 
under similar rationale. 

As to claim 28, it is individually similar in scope to claim 13 above; therefore, rejected 
under similar rationale. 

As to claim 30, it is individually similar in scope to claim 16 above; therefore, rejected 
under similar rationale. 

Claim Rejections - 35 USC § 103 

5. The following is a quotation of 35 U.S.C. 103(a) v/hich forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

6. Claim 2, 3, 8, 11, 17-21, 23, and 31-35 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Kuzunuki et al. (U.S. Patent No. 5,917,490) in view of Pierre Wellner ("The 
Digital Desk Calculator," Proceedings of ACM Symposium on User Interface Software and 
Technology (UIST '91), 11-13 Nov. 1991). 

As to claim 2, Kuzunuki teaches the information input/output system according to claim 
1, wherein at least one of said physical objects is capable of being moved in said information 
space and exchanging digital objects with others (note the rejection of claim 1 above); however, 
Kuzunuki does not teach that the physical object is a portable computer capable of being moved 
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in said information space and exchanging digital objects with other computers. An electronic 
object (physical object) of Wellner clearly discloses concept of controlling/interacting with the 
electronic object by scanning, and then projecting it back on an operating interface (see page 30 
column 1-2). Moreover, Wellner' s invention is not only limited that the electronic object is a 
DigitalDesk Calculator but also meant the same concept of using the electronic object as 
mentioned above can be applied on any regular PC if it gets scanned and projected on the 
operating interface ("the DigitalDesk Calculator is an example that illustrates how merging these 
two worlds can improve the usability of a very simple and well-established application," sic, see 
Conclusion on page 32). It would have been obvious at the time of the invention that a person 
whh ordinary skill in the art would want to have the system allows the user to interact with the 
electrical object of Wellner in the environment of Kuzunuki to share computer functionality to 
the physical objects and environment while allowing users to continue interacting with the 
objects as before (column 1, page 32). 

As to claims 3 and 23, they are similar in scope to claim 2 above; therefore, rejected 
under similar rationale. 

As to claim 8, it is similar in scope to claim 2 above; therefore, rejected under similar 
rationale. 

As to claim 1 1, it is similar in scope to claim 2 above; therefore, rejected under similar 
rationale. 

As to claim 17, Kuzunuki in view of Wellner teaches an information input/output system 
to be used for user operations relating to an object in an information space realized by expanding 
a digital space of a computer into the real world, said system comprising: 
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one or more than one operation surfaces arranged in said information space; a display 
means for displaying an image on said operation surfaces; an imaging means for picking up an 
image of said operation surfaces; a stationary environment type computer arranged in said 
inforrhation space; one or more than one portable computers capable of exchanging digital 
objects with other computers; and one or more than one physical objects mounted on said 
operation surfaces and containing a visually identifiable visual marker on the surface thereof 
(note the rejection of claim 12 above); 

said environment type computer being adapted to allow mouse/cursor operations on said 
portable computers to extend to said operation surfaces and display a rubber band as visual 
feedback (actual object handling, col. 9 lines 6-29) according to the area on said operation 
surfaces as defined by means of said mouse/cursor (note the rejection of claim 2 above). 

As to claim 18, it is similar in scope to claim 17 above; therefore, rejected under similar 
rationale. 

As to claim 19, Kuzunuki teaches the information input/output system according to claim 
17, wherein another object can apply a process or a method it possesses to the rubber-banded 
original digital object in response to an action of said original object of being dropped on said 
another object (actual object handling, col. 9 lines 6-29, and figs. 6A-8B). 

As to claim 20, Kuzunuki teaches the information input/output system according to claim 
19, wherein said another object is a physical object possessing a process or a method for printing 
a document (printers, col. 15 lines 1-10). 

As to claim 21, Kuzunuki teaches the information input/output system according to claim 
19, wherein said another object is a name card possessing a process or a method for mailing a 
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document to the mail address corresponding to the name on it (e.g., col. 8 lines 15-31, and fig. 
3A). 

As to claim 31, it is individually similar in scope to claim 17 above; therefore, rejected 
under similar rationale. 

As to claim 32, it is individually similar in scope to claim 18 above; therefore, rejected 
under similar rationale. 

J^s to claim 33, it is individually similar in scope to claim 19 above; therefore, rejected 
under similar rationale. 

As to claim 34, it is individually similar in scope to claim 20 above; therefore, rejected 
under similar rationale. 

As to claim 35, it is individually similar in scope to claim 21 above; therefore, rejected 
under similar rationale. 

Response to Arguments 

7. Applicant's arguments, see Amendment A, filed 04/20/2004, with respect to the 
rejection(s)of claim(s) 1-13, 15-28, and 30-35 under 102 rejections have been fully considered 
and are persuasive. Therefore, the rejection has been withdrawn. However, upon further 
consideration, a new ground(s) of rejection is made in view of Kuzunuki et al. (U.S. Patent No. 
5,917,490), and further in view of Pierre Wellner ("The Digital Desk Calculator," Proceedings of 
ACM Symposium on User Interface Software and Technology (UIST '91), 1 1-13 Nov. 1991). 
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Conclusion 

8. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to True T Chuong whose telephone number is 703-305-5753. The 
examiner can normally be reached on M-Th and alternate Fridays 8:30 AM - 5:00 PM, 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor. Heather R. Herndon can be reached on 703-308-5186. The fax phone number for the 
organization where this application or proceeding is assigned is 703-872-9306. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 



True T. Chuong 

07/20/04 



